Challenges in evaluating complex IT security management systems

Performing ecologically valid user studies for IT security management (ITSM) systems is challenging. The users of these systems are security professionals who are difficult to recruit for interviews, let alone controlled user studies. Furthermore, evaluation of ITSM systems inherits the difficulties of studying collaborative and complex systems. During our research, we have encountered many challenges in studying ITSM systems in their real context of use. This has resulted in us investigating how other usability evaluation methods could be viable components for identifying usability problems in ITSM tools. However, such methods need to be evaluated and proven to be effective before their use. This paper provides an overview of the challenges of performing controlled user studies for usability evaluation of ITSM systems and proposes heuristic evaluation as a component of usability evaluations of these tools. We also discuss our methodology for evaluating a new set of usability heuristics for ITSM and the unique challenges of running user studies for evaluating usability evaluation methods.